This paper takes a first step in aiding researchers to improve the relevance of their research to practice. By proposing that Information Systems researchers conduct applicability checks with practitioners on the research objects (for example, theories, models, frameworks, processes, technical artifacts, or other theoretically based IS artifacts) they either produce or use in theory-focused research, our paper presents an actionable, systematic approach to evaluating, establishing, and further improving research relevance. Furthermore, because it is an approach that can be conducted as an additional step either at the beginning or the end of the traditional research life cycle, it leaves untouched the rigorous methods used to conduct the study, that is, it does not compromise traditional research models. The approach we propose is based on the analyses of three dimensions of relevance that are critical to practitioners' attempts to internalize IS research findings (importance, accessibility, and suitability), and a comprehensive set of solutions that can be used to address them. Our analysis reveals that the most critical dimension for practice is the importance of the research to the needs of practice. The solution we propose to address that need is to conduct an applicability check on the research objects of interest. The applicability check forms an integral part of the research process, either prior to or following engagement in a typical research process. We present principles and criteria for the conduct and evaluation of an applicability check, which is primarily based on the focus group method, and secondarily on a modified nominal group technique.
Although information systems (IS) problem solving involves knowledge of both the IS and application domains, little attention has been paid to the role of application domain knowledge. In this study, which is set in the context of conceptual modeling, we examine the effects of both IS and application domain knowledge on different types of schema understanding tasks: syntactic and semantic comprehension tasks and schema-based problem-solving tasks. Our thesis was that while IS domain knowledge is important in solving all such tasks, the role of application domain knowledge is contingent upon the type of understanding task under investigation.We use the theory of cognitive fit to establish theoretical differences in the role of application domain knowledge among the different types of schema understanding tasks. We hypothesize that application domain knowledge does not influence the solution of syntactic and semantic comprehension tasks for which cognitive fit exists, but does influence the solution of schema-based problem-solving tasks for which cognitive fit does not exist.To assess performance on different types of conceptual schema understanding tasks, we conducted a laboratory experiment in which participants with high- and low-IS domain knowledge responded to two equivalent conceptual schemas that represented high and low levels of application knowledge (familiar and unfamiliarapplication domains). As expected, we found that IS domain knowledge is important in the solution of all types of conceptual schema understanding tasks in both familiar and unfamiliar applications domains, and that the effect of application domain knowledge is contingent on task type. Our findings for the EER model were similar to those for the ER model. Given the differential effects of application domain knowledge on different types of tasks, this study highlights the importance of considering more than one application domain in designing future studies on conceptual modeling.
Although there is a long tradition of empirical studies of software developers, few studies have focused on software maintenance. Prior work is predicated on the belief that higher levels of software comprehension are associated with higher levels of performance on modification tasks. This study provides a more complete understanding of the relationship between software comprehension and modification. We conceptualize software maintenance as interlinking comprehension and modification, and argue that the relationship between the two is moderated by cognitive fit. Specifically, cognitive fit exists when the software maintainer's dominant mental representation of the software and their mental representation of the modification task emphasize the same type of knowledge. We hypothesize that when cognitive fit exists, greater improvements in comprehension are associated with higher levels of performance on a modification task. When cognitive fit does not exist, however, the software maintainer's mental representations of the software and of the modification task do not emphasize the same type of knowledge, which may mean that attention is devoted to comprehension at the expense of modification, resulting in lower performance on the modification task. In these circumstances, comprehension and modification tasks may interfere with each other, an effect known as dual-task interference. We therefore hypothesize that performance on a modification task is moderated by the fit between the mental representation of the software and that of the modification task. We tested our theory by varying cognitive fit to create matched and mismatched conditions in a single experiment that used IT professionals as subjects. Our findings support our theory: cognitive fit moderates the relationship between comprehension and modification. Specifically, changes in software comprehension and modification performance are positively related when cognitive fit exists and negatively related when cognitive fit does not exist. Our findings demonstrate the need to examine more complex relationships among the numerous types of tasks involved in software development rather than examining software comprehension alone.
Throughout its history, the information systems (IS) discipline has engaged in extensive self-examination, particularly with regard to its apparent diversity. Our overall objective in this study is to better understand the diversity in IS research, and the extent to which diversity is universal across journals that publish IS research. We developed a classification system that comprises five key characteristics of diversity (reference discipline, level of analysis, topic, research approach, and research method) based on a review of prior literature. We then examined articles over a five-year period, from 1995 to 1999, in five journals acknowledged as the top journals of the field, at least in North America. Analyses reveal considerable diversity in each of the key characteristics. Perhaps not surprisingly, the research approach used is more focused with most studies being conducted using hypothetico-deductive approaches, whereas reference discipline is perhaps the most diverse of the characteristics examined. An interesting finding is that IS itself emerged as a key reference discipline in the late 1990s. The 'Journal of Management Information Systems' and 'Information Systems Research' publish articles displaying the greatest diversity, and 'MIS Quarterly' and 'Decision Sciences' publish articles that focus on subsets of the field. Our research provides a foundation for addressing the direction that diversity in the IS discipline takes over time. In the shorter term, researchers can use our classification system as a guide to writing abstracts and selecting key words, and the findings of our journal analyses to determine the best outlet for their type of research.
Recent research using professional programmers suggests that knowledge of the application domain plays a major role in the cognitive processes they use to understand computer programs. In general, programmers use a more topdown comprehension process when working in familiar application domains, and a more bottom-up process in unfamiliar domains. The present study builds on that research by further characterizing comprehension processes. The findings show that: (1) certain programmers use different types of comprehension processes depending on their familiarity with the application domain (flexible approach), while others do not (top-down and bottom-up approaches); (2) familiarity with the application domain and the use of a particular comprehension process have marked effects on references programmers make to both application and programming domain knowledge; and (3) programmers who use a flexible comprehension process achieved the highest levels of comprehension. The present research also examines some cognitive determinants of the comprehension process. The findings highlight the need to consider application, as well as programming, domain knowledge as areas of computer programming expertise, to investigate factors influencing use of specific comprehension processes, and to develop tools to support flexible comprehension processes.
The field of software, has, to date, focused almost exclusively on application-independent approaches. In this research, we demonstrate the role of application domain knowledge in the processes used to comprehend computer programs. Our research sought to reconcile two apparently conflicting theories of computer program comprehension by proposing a key role for knowledge of the application domain under examination. We argue that programmers use more top-down comprehension processes when they are familiar with the application domain. When the application domain is unfamiliar, programmers use processes that are more bottom-up in nature. We conducted a protocol analysis study of 24 professional programmers comprehending programs in familiar and unfamiliar application domains. Our findings confirm our thesis.
The systems development process involves establishing the information requirements of an application and successively transforming those requirements into a computer-based model of the application. Attention is usually focused almost exclusively on the method of transformation, however, with little recognition of the role of the application. As a first step in examining the relevance of knowledge of the application to the systems development process, this study addresses whether there are synergistic effects of application and methodology knowledge in specifying information requirements. This was achieved via a repeated-measures protocol analysis study that manipulated both experience with the application and knowledge of the methodology. The results show that in learning to specify information requirements, novice analysts: performed more effectively over time when trained to use a methodology applied the methodology more effectively when familiar with the application; performed more effectively when they used procedural methodology knowledge rather than declarative methodology knowledge alone; improved the effectiveness of their problem solving over time only when they used procedural methodology knowledge; produced idiosyncratic results based on the application. Based on the findings of this research, it appears that research into the nature of applications, as well as methodologies, is warranted. From the viewpoint of the practitioner, since application knowledge is idiosyncratic, it may be necessary to include more than one application-knowledgable person on a systems development team.
From a broad perspective, our research can be viewed as investigating the fit of technology to task, the user's view of the fit between technology and task, and the relative importance of each to problem-solving or decision-making performance. The technology investigated in this research is the mode of information presentation. Although there has been a considerable amount of research into problem solving using graphs and tables, until recently the circumstances in which each is more effective have been largely unresolved. Recent research has suggested that performance benefits accrue when cognitive fit occurs, i.e., when factors such as the problem representation and problem solving tools match the characteristics of the task. In this paper, we investigate the effects of the basic paradigm of cognitive fit and extensions to the paradigm in a laboratory experiment that examined the nature of subjects' mental representations as well as problem-solving performance. The experiment, using 128 MBA students in two identical, repeated measures designs, produced the following results: • Performance improved markedly for symbolic tasks when the problem representation matched the task • Performance effects also resulted from matching specific problem-solving skills to the problem representation and the task, and to a lesser extent when the skills matched the task alone. • The incremental effects of matching skills to the problem representation and/or the task were small compared with the primary effects of cognitive fit-that of matching problem representation to task. • A large proportion of problem solvers have insight into the concept of supporting tasks with certain types of problem representation and vice versa. • Participants preferred to use tables rather than graphs; they also preferred to solve symbolic rather than spatial problems. • Finally, the problem representation more significantly influenced the mental representation than did task conceptualization. This research suggests that providing decision support systems to satisfy individual managers' desires will not have a large effect on either the efficiency or the effectiveness of problem solving. Designers should, instead, concentrate on determining the characteristics of the tasks that problem solvers must address, and on supporting those tasks with the appropriate problem representations and support tools. Sufficient evidence now exists to suggest that the notion of cognitive fit may be one aspect ofa general theory of problem solving. Suggestions are made for extending the notion of fit to more complex problem-solving environments.
Despite the fact that commercial computer systems have been in existence for almost three decades, many systems in the process of being implemented may be classed as failures. One of the factors frequently cited as important to successful system development is involving users in the design and implementation process. This paper reports the results of a field study, conducted on data from forty-two systems, that investigates the role of user involvement and factors affecting the employment of user involvement on the success of system development. Path analysis was used to investigate both the direct effects of the contingent variables on system success and the effect of user involvement as a mediating variable between the contingent variables and system success. The results show that high system complexity and constraints on the resources available for system development are associated with less successful systems.
Information Systems (IS) is facing a dilemma: software is absorbing an ever-increasing proportion of the total IS budget while maintenance is absorbing an ever-increasing proportion of the software budget. In the not too distant future, unless this trend is arrested, or reversed, nearly all software resources may be required for maintenance. There are certain projected developments which give hope for the future -- though not in the short term. In the meantime IS management must use other approaches, approaches that are available today, but that have been adopted very slowly. This article discusses one viable approach -the use of automated programmer productivity tools. It presents a classification of programmer productivity tools presently on the market and outlines the ways in which each class addresses productivity. It also discusses the important considerations IS management must make when evaluating the adoption of such tools.
The ability to estimate the personnel time and costs required for the completion of programming and systems projects is an important managerial tool for the information systems department. This article presents a survey of the estimation techniques found in the literature by describing each technique and discussing its strengths and weaknesses. Some empirical evidence on how the various program and programmer/analyst characteristics affect project time and cost are also reported.